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Field of Invention 



[0001] This invention presents a set of methods and apparatus for omnidirectional stereo £^ 
imaging. By "omnidirectional imaging system", we mean a system that is able to acquire^ 
images with a fleld-of-view covering entire hemisphere (180 solid space angle), 
simultaneously without any mechanical moving part. The field of view of a conventional 
camera or a light projector can be dramatically increased by employing a reflective < c^ 
mirror properly placed in front of the camera or the projector. A pair of omnidirectiona$l. cp^ 
cameras is able to form a unique stereo imaging system that is able to obtain three 
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dimensional images of surrounding scene with 360 degree view angle. A combination of r£ 
an omnidirectional camera and an omnidirectional structured light projector can also r ,p *~ 

provide means to obtain quantitative three dimensional measurements of the objects 
around the camera system. The omnidirectional three dimensional imaging methods and 
apparatus presented herein may offer unique solutions to many practical systems that 
need simultaneous 360 degree viewing angle and three dimensional measurement 
capability. 



Prior Art Existing A pproaches to Large FQV Imag in g System 
[0002] A number of approaches had been proposed in the past for imaging systems to 
achieve wide field-of-view (FOV). None of them, however, is able to generate 3D 
omnidirectional images. In the following paragraphs, we give a briefly survey on the 
state-of-the-art of current imaging systems that seek to achieve wide FOV. 

Conventional Cameras 
[0003] Most existing imaging systems employ electronic sensor chips, or still 
photographic film, to record optical image collected by its optical lens system. The 
image projection for most camera lenses is modeled as a "pin-hole" with a single center 
of projection. Since sizes of camera lens and the imaging sensor have their practical 
limitations, the light rays that can be collected by a camera lens and received by the 
imaging device typically form a corn with very small opening angle. Therefore, angular 



field-of-views for conventional cameras are within a range of 5 to 50 degrees. For 
example, an 8.5 mm F/1.3 camera lens for V% CCD (Charge Coupled Device) chip only 
has an angular FOV of 41 .2 degree. 

Fish-Eye Lenses 

[0004] Optical engineers had designed several versions of wide-viewing-angle lens 
system, called the fish-eye lens (see [1],[2]). The fish-eye lens features a very short focal 
length which, when used in place of conventional camera lens, enables the camera to 
view object for much wider angle (almost 180 degree of hemisphere). In general, the 
wider FOV, the more complicated design the fish-eye lens has. To obtain a 
hemispherical FOV, the fish-eye lens must be quite large in dimension, complex in 
optical design, and hence expensive. Also, it is very difficult to design a fish-eye lens 
that ensures single view point constraint, i.e., all incoming principal light rays intersect at 
a single point to form a fixed viewpoint. This is indeed a problem with commercial fish- 
eye lenses including Nikon's Fisheye-Nikkor 8 mm f/2.8 lens. The use of fish-eye lenses 
for wide FOV imaging application has been advocated by [3] and [4], among others. 
Although the acquired image by fish-eye lenses may prove to be good enough for some 
visualization applications, the distortion compensation issue has not been resolved, and 
the high unit-cost remain to be major hurdles for its wide-spread applications. The fish- 
eye lens technique has the advantage of adopting a statically positioned camera to acquire 
a wide angle view. However the nonlinear property resulted from the semi-spherical 
optical lens mapping make the resolution along the circular boundary of the image very 
poor, while the field of view corresponding to the circular boundary of the image usually 
represents a ground or floor where a high resolution of image is required. 

Multi-Camera System or Rotating Imaging System 
[0005] Large field of view of objects may be obtained by using multiple cameras in the 
same system, each points towards a different direction. However, issues on seamless 
integration of multiple images is further complicated by the fact that image produced by 
each camera has different centers of projection. The cost for such a system is usually 
high. The image processing required by multiple cameras or rotating camera method to 



obtain a precise information on position and azimuth of an object takes a long time, 
which is not suitable for real-time battle field modeling and reconnaissance applications. 
[0006] Another straightforward solution to increasing the FOV of an imaging system is 
to rotate the entire imaging system about its center of projection (Figure 1). An image 
sequence acquired by the camera at different positions are "stitched" together to obtain a 
panoramic view of the scene. Such an approach has been recently proposed by several 
researchers (see [5], [6] [7]). A very interesting approach developed by [8] employs a 
camera with a non-frontal image detector to scan the world. 

[0007] The first disadvantage of any rotating image system is that it requires the use of 
moving parts, and precision positioning devices. A more serious drawback is that such 
systems lack the capability of simultaneously acquiring images with wide FOV. 
Although such system can acquire precise azimuth information in omnidirectional view, 
the imaging process is time-consuming and the method is not applicable to real-time 
problems such as avoiding collision against moving obstacles or monitoring scene with 
mobile objects. This restricts the use of rotating systems to static and non-real-time 
applications, 

[0008] In contrast, the invention presented herein, called the omnidirectional camera, is 
capable of capturing real-time omnidirectional images without using any moving parts. 
By "omnidirectional images", we mean images with a field-of-view covering entire 
hemisphere (180 solid space angle), simultaneously. Figure 2 provides a comparison 
between our Omnidirectional Camera, panoramic camera and conventional cameras. As 
one can see, a panoramic camera is still not omnidirectional, since it can only provide a 
wide-angle of FOV at certain time instance, not in all directions. 

Summary of the Invention 
[0009] The primary objective of the present invention is to provide a set of simple 
methods and apparatus to obtain simultaneously omnidirectional stereo images without 
using any moving parts. The field of view of a conventional camera or a light projector 
can be dramatically increased by employing a reflective mirror properly placed in front o 
the camera or the projector. A pair of omnidirectional cameras is able to form a unique 
stereo imaging system that is able to obtain three dimensional images of surrounding 




scene with 360 degree view angle. A combination of an omnidirectional camera and an 
omnidirectional structured light projector can also provide a means to obtain quantitative 
three dimensional measurements of the objects around the camera system. The 
omnidirectional three dimensional imaging methods and apparatus presented herein may 
offer unique solutions to many practical systems that need simultaneous 360 degree 
viewing angle and three dimensional measurement capability. 

Brief Description of the Drawing s 
[0010] Figure 1 shows a conventional way to obtain panoramic images; the composition 
of multiple views of a rotated camera into a panoramic image. 

[0011] Figure 2 gives a comparison of the Field of View (FOV) among conventional 

cameras, panoramic cameras, and the proposed omnidirectional camera. 

[0012] Figure 3 provides examples of reflective convex mirror for Omni-Directional 

Imaging. Notice that these convex mirrors do not satisfy the single viewpoint constraint 

(SVC) condition. The (extension of) reflected rays do not meet at single viewpoint, i.e., 

the virtual viewpoint varies with rays' impinging location on the mirror. 

[0013] Figure 4 acquires Omni-Directional Image from the Omni-Mirror. A video 

camera place at location C can "see" objects in an entire hemisphere FOV, from a single 

virtual viewpoint at mirror's focal center O. 

[0014] Figure 5 shows an embodiment of the Omnidirectional 3D Camera. 
[0015] Figure 6: Circular Variable Wavelength Filter (CVWF). 

[0016] Figure 7: Omnidirectional Rainbow Light Projection Using the Omni-Mirror and 
CVWF. 

[0017] Figure 8: Omnidirectional Structured Light 3D Camera. 

Description of the Preferred Embodiments 
[0018] To dramatically increase the field of view of an imaging system, we propose a 
somewhat unusual approach: using a reflective surface (i.e., convex mirror). The field of 
view of a video camera can be greatly increased by using reflective surface with properly 
designed surface shapes. The rear-view mirror in a car is a daily example of using 
reflective mirror to increase the FOV of a driver. 




[0019] There are a number of surface profiles that can be used to produce 
omnidirectional FOV. Figure 3 lists three examples: conic mirror, spherical mirror, and 
parabolic mirror. The optical geometry of these convex mirrors provides a simple and 
effective means to convert video camera's planar view into an omnidirectional view 
around the vertical axis of these mirrors, without using any moving part. 
[0020] At the first glance, it appears that the omnidirectional imaging task can be 
accomplished by using any convex mirror. Unfortunately, this is not the case. In 
reviewing some basic of image formation, we know that an image is a two dimensional 
pattern of brightness (or colors) . A satisfactory imaging system must preserve two 
essential characteristics: 

(1) Geometric correspondence: there must be a one-to-one correspondence 
between pixels in an image and point in the scene. 

(2) Single Viewpoint Constraint: each pixel in the image corresponds to a 
particular viewing direction defined by a ray from that pixel on image plane through a 
"pinhole" (single viewing point). 

[0021] Notice that, although the convex mirrors listed in Figure 3 can greatly increase the 
FOV, and may prove adequate for certain omnidirectional scene monitoring applications, 
they are not satisfactory imaging devices. These reflecting surfaces do not preserve the 
single viewpoint constraint (SVC). For a high quality omnidirectional imaging system, 
all the light rays coming in the omni imager head should have a single (virtual) viewing 
point. 

Design of the Omni-Mirror That Meets the SVC 
[0022] In this section, we will discuss a desirable convex mirror surface profile that 
satisfies the single viewpoint constraint: all the (extensions of) light rays reflected by the 
mirror must pass through a single (virtual) viewpoint. We call such a reflective mirror 
the OMNI -mirror. 




[0023] Let us first define necessary symbols and terminology. As shown in the Figure 4, 
we use an off-the-shelf video camera with a regular lens whose FOV covers entire 
surface of the OMNI-mirror. Since the optical design of camera and lens is rotationally 
symmetric, all we need to determine is the cross-section function z(r) that defines the 
mirror surface cross section profile. The mirror is then the solid of revolution obtained 
by sweeping the cross-section about the optical axis. The function of the omni-mirror is 
to reflect all viewing rays coming from video camera's viewing center (focal point, 
labeled as C) to the surface of physical objects in the field-of-view. The key feature of 
this reflection is that all such reflected rays must have a projection towards a single 
virtual viewing point at mirror's focal center, labeled as O. In other words, the mirror 
should effectively steer viewing rays such that the camera equivalently sees the objects in 
the world from a single viewpoint O. 

[0024] We choose hyperbolic as the desirable shape of the Omni-mirrors. A well-known 
feature of hyperbolic curve is that: the extension of any ray reflected by the hyperbolic 
curve originated from one of its focal points passes through its another focal point. If we 
choose the hyperbolic the hyperbolic profile for the OMNI-mirror, and place a video 
camera at its focal point C, as shown in Figure 4, the imaging system will have a single 
viewpoint at its another focal point O, as if the video camera were placed at the virtual 
viewing location O. 

[0025] The mathematical equation that describes the hyperbolic Omni-Mirror surface 
profile is: 

(z + c) V = ^ where c = J^ + b 2 and f = 2f , (1) 
b'a 

The unique feature of the Omni-Mirror is that the extension of the incoming light ray 
sensed by the CCD camera is always passing through a single virtual viewpoint O 
regardless of the location of the projection point M on the mirror surface. 



Acquiring Image Using an Omni Mirror 
[0026] For any point P in the scene, the image reflected from the OMNI mirror to image 
sensor's image plane has the radius of d c : 




where u and v are the pixel indexes on the image plane. As shown in Figure 4, the 
camera viewing angle corresponding to the point on the scene is given by: 

rc = tan-'^ (3) 

J C 

The incoming light ray to the camera can be described by a line equation: 

z=— / (4) 

tan y c 

Omitting details of mathematical derivation based on the OMNI-Mirror equation (1), we 
can obtain a simple closed-form relationship between the omnidirectional viewing angle 
a and CCD camera's viewing angle y c : 

2bc-(b 2 + c 2 )cosy c 



a 2 sin y c 



(5) 



[0027] This equation establishes a one-to-one corresponding relationship between a and 
y c . This relationship is important when we perform triangulation calculation in the 
omnidirectional 3D camera system. 

First Embodiment: Omnidirectional Stereo Camera (QSC) 
[0028] With a pair of the omnidirectional cameras, we can easily construct an 
omnidirectional stereo camera. Figure 5 shows a possible configuration of the OSC. 
Two omni-mirrors are placed face-to-face with optical axes aligned. The virtual imaging 
centers, Oiand O2, are separated by a distance of B, which forms the baseline for the 
stereo vision. 

[0029] The triangulation can be carried out directly from omnidirectional images without 
the need for image conversion. Once a 3D object is detected in the omnidirectional 
image, the viewing angle y x and y 2 can be determined from cameras' geometry. Based 
on equation (5), the virtual viewing angles of this 3D object, a x and a 2 from the virtual 
viewpoint Oiand 0 2 can be determined. The distance between one of the viewing center 



and the 3D object in the scene can be calculated using straight forward triangulation 
principal: 

R= C ° S( ^> B (6) 

sin(^ + a 2 ) 

where R is the Range, i.e. distance between P and Oi. 

Second Embodiment: Omnidirectional Structured Light 3D Camera 

Projecting Structured Illumination Using the Omni-Mirror Circular Variable 

Wavelength Filter TCVWF) 

[0030] Notice that in our Omni 3D camera system, a circular variable wavelength filter 
(CVWF) is used to facilitate the generation of 360° projection illumination through a 
convex mirror. 

[0031] The CVWF is a circular optical glass plate coated with color rings of gradually 
varying wavelengths within visible (400 - 700 nm) or IR (> 700 nm) region. The 
wavelengths of the coated color rings are linearly proportional to their radians measured 
from the center of the filter glass. This feature provides a simple and elegant way of 
generating structured light for an entire scene without using any moving parts (Figure 6). 
[0032] The wavelength of the light A passing a particular position of the CVWF is a 
linear function of r , the radians measured from the center of the filter glass: 

Mr) = JL e+ ^^r (7) 
K 

where A cJ A a and R are filter parameters: A c is the wavelength corresponding to the 
filter's "center" (lowest wavelength color the filter can generate). X a is the wavelength 
corresponding to the filter's "edge" (highest wavelength color the filter can generate). R 
is the effective radian of the filter's "edge". And r is the radians measured from the 
center of the filter glass. 

[0033] If the relative position of the CVWF is fixed with respect to the white light 
projector, the projected light with the same wavelength forms light sheets of circular 
conic shapes. The cone angle 0 between the light sheet and the normal line of the 
projector is directly related to the wavelength of the light. In other word, the wavelength 



of the light is encoded with the geometric information of the projected corn angle. This 
fixed wavelength Xto angle 0 relationship is the key idea for our rainbow 3D camera 
system. 



Qmni-directional Rainbow Light Projection Using the QMNI-Mirror 
[0034] In order to obtain a Rainbow-like illumination with 360° omnidirectional 
projecting angle, we use an Omni-Mirror to reflect the conic-shaped light sheets 
generated by the white light projector and a CVWF. As shown in Figure 7, the projection 
system has a 360° effective projecting angle around the vertical axis of the OMNI mirror. 
[0035] The OMNI-Mirror mapping provides an easy and elegant way to determine the 
projecting angle 0 of a conic light sheet based on the projection angle of the white light 
projector (WLP) y . The projection line equation is given by: 

/ (8) 



tan/ 

Combining with the OMNI-Mirror equation 



(Z 2 C)2 -4-- 1, where c = ^a 2 +b 2 (9) 
b a 

the reflecting point (r mp , z mp ) on the mirror can be calculated by solving a quadratic 
equation. Using some properties of the OMNI-Mirror and simplification procedures in 
mathematical manipulations, the relationship between the white light projection angle y , 
and the omni -projection angle 9 can then be expressed as: 

g = tan- 2bc -( b l +cl )™r (1Q) 
a sin/ 

In other words, knowing the white light projection angle / and parameters of the OMNI- 
Mirror, the omni-projection angle 0 is fully determined. Since the value of /determines 
the projected wavelength in the rainbow spectrum X , spatial geometric characteristics of 
the projected cone shape Rainbow light sheet are fully defined. Although the 3D range 
calculation of the Omni-3D camera does not require the precise relationship between 0 
and X , such a concise relationship facilitates a simple design, implementation and tests 
of the omnidirectional rainbow light projector. 





Omnidirectional Structured Light 3D Camera 



Figure 8 shows an embodiment of an omnidirectional structured light 3D camera. 
An Omnidirectional rainbow light projector is used herein to produce a spatially varying 
wavelength illumination in the surrounding scene. An omnidirectional camera is placed 
co-axially (i.e., with optical axes aligned) with the omni-directional rainbow projector. 
The virtual projection centers, Oi, and the virtual imaging center O2, are separated by a 
distance of B, which forms the baseline for the triangulation based 3D vision system. 
The triangulation can be carried out directly from the omnidirectional images without the 
need for image conversion. Once a 3D object is detected in the omnidirectional image, 
the viewing angle yt is determined from cameras' geometry. The projection angle y \ is 
determined from the wavelength of the structured light projection, due to the one-to-one 
corresponding relationship of the Rainbow projector between the wavelength and the 
projection angle. The distance between the virtual imaging center O2 and the surface 
points on the 3D object in the scene can be calculated using straight forward triangulation 
principle: 



R = 



cos(ar 2 ) 



B 



(11) 



sin(^ + a 2 ) 



